NicolaSpreafico
commented
7 years ago Any update on this?
Closed elharo closed 4 years ago
NicolaSpreafico
commented
7 years ago Any update on this?
lm2a
commented
4 years ago Hi, IMO shouldn't be so complex to use GCS. Something goes wrong in its architecture. There are a lot of people having troubles with createOrReplace and Google simply doesn't take care to give us some support. In my case running exactly Google AppEngine GCS Client Example I am having IOException fetching block GcsServiceImpl.createOrReplace And really there is not a place to get any clue. Very bad, Google.
loosebazooka
commented
4 years ago @saturnism @ludoch any idea who would be responsible or this?
lm2a
commented
4 years ago No, but I am considering switch to Amazon EC
elharo
commented
4 years ago Have you tried updating to appengine-gcs-client 0.8? There were some problems in the dependencies prior to that.
We really need to either release this damn thing or get rid of it.
lm2a
commented
4 years ago Yes, I have
Still failing, and I think we deserve some support, we are paying to Google every month
elharo
commented
4 years ago If you have a support contract, please file a ticket through those channels. That should get attention from folks who can handle this.
saturnism
commented
4 years ago @lm2a can you share parts of your app.yaml? is it using app engine standard or flex? thanks,
lm2a
commented
4 years ago Thanks, @elharo but beyond this punctual problem, GCS looks messy. I mean there are different libraries doing the same without compatibility among them. And there isn't a clear difference to choose one or another. In my case I want to use GCS as a CDN, so I need to save and recover images and resizing it on the fly. There is an image API to do that, but it is just available on App Engine, but when I try to use appengine-gcs-client it fail and there is dead road. I was successful uploading objects using Rest API, but seems as it is not compatible with Image API.
lm2a
commented
4 years ago Flex @saturnism, here you have my yaml:
runtime: java env: flex
resources: cpu: 1 memory_gb: 1 disk_size_gb: 10
liveness_check: check_interval_sec: 120 timeout_sec: 40 failure_threshold: 5 success_threshold: 5 initial_delay_sec: 500
readiness_check: check_interval_sec: 120 timeout_sec: 40 failure_threshold: 5 success_threshold: 5 app_start_timeout_sec: 1500
saturnism
commented
4 years ago @lm2a to make sure i'm on the same page, can you share the sample code that you were looking at? Secondly, did you see the error after deployment, or when running locally?
lm2a
commented
4 years ago Hi, it is the Google Sample:
import com.google.appengine.api.blobstore.BlobKey;
import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;
//[START gcs_imports]
import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsInputChannel;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;
//[END gcs_imports]
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.Channels;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* A simple servlet that proxies reads and writes to its Google Cloud Storage bucket.
*/
@SuppressWarnings("serial")
public class GcsExampleServlet extends HttpServlet {
public static final boolean SERVE_USING_BLOBSTORE_API = false;
/**
* This is where backoff parameters are configured. Here it is aggressively retrying with
* backoff, up to 10 times but taking no more that 15 seconds total to do so.
*/
private final GcsService gcsService = GcsServiceFactory.createGcsService(new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build());
/**Used below to determine the size of chucks to read in. Should be > 1kb and < 10MB */
private static final int BUFFER_SIZE = 2 * 1024 * 1024;
/**
* Retrieves a file from GCS and returns it in the http response.
* If the request path is /gcs/Foo/Bar this will be interpreted as
* a request to read the GCS file named Bar in the bucket Foo.
*/
//[START doGet]
@Override
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
GcsFilename fileName = getFileName(req);
if (SERVE_USING_BLOBSTORE_API) {
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
BlobKey blobKey = blobstoreService.createGsBlobKey(
"/gs/" + fileName.getBucketName() + "/" + fileName.getObjectName());
blobstoreService.serve(blobKey, resp);
} else {
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, BUFFER_SIZE);
copy(Channels.newInputStream(readChannel), resp.getOutputStream());
}
}
//[END doGet]
/**
* Writes the payload of the incoming post as the contents of a file to GCS.
* If the request path is /gcs/Foo/Bar this will be interpreted as
* a request to create a GCS file named Bar in bucket Foo.
*/
//[START doPost]
@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
GcsFileOptions instance = GcsFileOptions.getDefaultInstance();
GcsFilename fileName = getFileName(req);
GcsOutputChannel outputChannel;
outputChannel = gcsService.createOrReplace(fileName, instance);
copy(req.getInputStream(), Channels.newOutputStream(outputChannel));
}
//[END doPost]
private GcsFilename getFileName(HttpServletRequest req) {
String[] splits = req.getRequestURI().split("/", 4);
if (!splits[0].equals("") || !splits[1].equals("gcs")) {
throw new IllegalArgumentException("The URL is not formed as expected. " +
"Expecting /gcs/<bucket>/<object>");
}
return new GcsFilename(splits[2], splits[3]);
}
/**
* Transfer the data from the inputStream to the outputStream. Then close both streams.
*/
private void copy(InputStream input, OutputStream output) throws IOException {
try {
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = input.read(buffer);
while (bytesRead != -1) {
output.write(buffer, 0, bytesRead);
bytesRead = input.read(buffer);
}
} finally {
input.close();
output.close();
}
}
}
I was changing code and now I am having other similar error, here you have the logs:
Caused by:
java.lang.NullPointerException
at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.
elharo
commented
4 years ago I've moved this to the appengine-gcs-client repo which is where the root cause more likely lies. See https://github.com/GoogleCloudPlatform/appengine-gcs-client/issues/81
ludoch
commented
4 years ago Hi, env: flex does not support at all the com.google.appengine.api* APIs, and I see a import com.google.appengine.api.blobstore.BlobKey; statement, so it will never work, and not documented to work. Secondly, com.google.appengine.tools.cloudstorage from the repo above is also version 0.8 and not a supported GA product.
Please use the GA, Supported Cloud Java APIs from: https://github.com/googleapis/google-cloud-java that work on GAE Standard and Flex and are fully supported. It includes a GCS library, see https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-clients/google-cloud-storage
lm2a
commented
4 years ago Hello, folks, many thanks for your kindly help. Finally I was able to create a service to store and resize images.
By the way, @elharo nice to meet you here, in some way you was my Java teacher, because I was learning a lot of Java from your books
Thanks!
IMO, is worth to leave some hints here can help to other people trying to use GCS as CDN with Java to keep and resize images on the fly. There is an example provided by Google This example should be running locally in your computer as App Engine (you had to install app engine Eclipse plugin before). (1) That example read and transform a file located under /WEB-INF If you try to run it using Jetty Eclipse plugin it will be failing to locate files in your /WEB-INF. Just do (1). I made an adaptation of that servlet to upload images using a Post. Here is the code:
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.nio.ByteBuffer;
import javax.servlet.ServletException;
import javax.servlet.annotation.MultipartConfig;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.Part;
import org.apache.commons.io.IOUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import com.google.appengine.api.images.Image;
import com.google.appengine.api.images.ImagesService;
import com.google.appengine.api.images.ImagesServiceFactory;
import com.google.appengine.api.images.ServingUrlOptions;
import com.google.appengine.api.images.Transform;
import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;
// [START example]
@SuppressWarnings("serial")
@MultipartConfig
// With @WebServlet annotation the webapp/WEB-INF/web.xml is no longer required.
@WebServlet(name = "images",
description = "Images: Write an image to a bucket and display it in various sizes",
urlPatterns = "/images")
public class ImagesServlet extends HttpServlet {
final String bucket = "lm2a-images-store";
// [START gcs]
private final GcsService gcsService = GcsServiceFactory.createGcsService(new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build());
// [END gcs]
@Override
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException,
IOException {
req.getRequestDispatcher("/form.jsp").forward(req, resp);
}
// [START formpost]
@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException,
IOException {
Part filePart = req.getPart("file");
final String fileName = filePart.getSubmittedFileName();
//String imageUrl = req.getParameter("imageUrl");
InputStream filecontent = filePart.getInputStream();
byte[] imageBytes = IOUtils.toByteArray(filecontent);
DateTimeFormatter dtf = DateTimeFormat.forPattern("-YYYY-MM-dd-HHmmssSSS");
DateTime dt = DateTime.now(DateTimeZone.UTC);
String dtString = dt.toString(dtf);
final String newFileName = dtString + fileName;
// Write the original image to Cloud Storage
gcsService.createOrReplace(
new GcsFilename(bucket, newFileName),
new GcsFileOptions.Builder().mimeType("image/jpeg").build(),
ByteBuffer.wrap(imageBytes));
//[END original_image]
//[START resize]
// Get an instance of the imagesService we can use to transform images.
ImagesService imagesService = ImagesServiceFactory.getImagesService();
// Make an image directly from a byte array, and transform it.
Image image = ImagesServiceFactory.makeImage(imageBytes);
Transform resize = ImagesServiceFactory.makeResize(100, 50);
Image resizedImage = imagesService.applyTransform(resize, image);
// Write the transformed image back to a Cloud Storage object.
gcsService.createOrReplace(
new GcsFilename(bucket, newFileName+"_"+"resizedImage.jpeg"),
new GcsFileOptions.Builder().mimeType("image/jpeg").build(),
ByteBuffer.wrap(resizedImage.getImageData()));
//[END resize]
ServingUrlOptions options = ServingUrlOptions.Builder
.withGoogleStorageFileName("/gs/" + bucket + "/"+newFileName)
.imageSize(150)
.crop(true)
.secureUrl(true);
String url = imagesService.getServingUrl(options);
// [END servingUrl]
// Output some simple HTML to display the images we wrote to Cloud Storage
// in the browser.
PrintWriter out = resp.getWriter();
out.println("<html><body>\n");
out.println("<img src='//storage.cloud.google.com/" + bucket
+ "/image.jpeg' alt='AppEngine logo' />");
out.println("<img src='//storage.cloud.google.com/" + bucket
+ "/resizedImage.jpeg' alt='AppEngine logo resized' />");
out.println("<img src='//storage.cloud.google.com/" + bucket
+ "/rotatedImage.jpeg' alt='AppEngine logo rotated' />");
out.println("<img src='" + url + "' alt='Hosted logo' />");
out.println("</body></html>\n");
}
// [END formpost]In that example the appengine-gcs-client has a dependency with servlet API 2.5 which convey to problems to use HttpServletRequest getPart(String x) method. You can solve it adding the right dependency and excluding the bad one:
<dependency>
<groupId>com.google.appengine.tools</groupId>
<artifactId>appengine-gcs-client</artifactId>
<version>0.8</version>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
Reported against the Eclipse-plugin here:
https://github.com/GoogleCloudPlatform/gcloud-eclipse-tools/issues/599
However it appears to at least be a bug in the maven gcloud plugin and may be a bug in gcloud itself.